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1. Introduction 

The Ashkin- Teller neural network (ATNN) studied recently by us in the case of loading 
of a finite number of patterns ([^ and references therein) can be considered as a non- 
trivial generalisation of the Hopfield model 0, ^ to allow for different types of neurons. 
It can be seen as a model consisting of two Hopfield models interacting through a four- 
neuron term, or it can be interpreted as a neural network with two types of neurons at 
each site having different functions. 

Some of the underlying neurobiological motivation for the introduction of different 
types of neurons is the fact that there exist areas in the brain which react to two 
different kinds of dependent stimuli in such a way that the response to particular 
combinations of these stimuli is stronger than the response to others [Q. Furthermore, 
in neuropsychological studies on amnesia it has become appreciated that memory is 
composed of multiple separate systems which can store different types of information, 
e.g., information based on skills and information based on specific facts or data 0. In 
addition, the classical Ashkin- Teller model on which our ATNN model is based, appears 
in studies of disordered systems when the disorder evolves on a time scale that can be 
tuned 0. Finally, two different types of neurons show up naturally when considering 
the Hopfield model with synchronous dynamics [|, 0, |]. 

Storage and retrieval of embedded patterns built from these two types of neurons 
with different degrees of (in) dependence have been discussed in [^ for finite loading, 
i.e. the capacity a, defined as the number of embedded patterns, p, versus the size 
of the system, N, is taken to be zero. In that study we wanted to find out, e.g., 
whether the four-neuron interaction between the two types of neuron can improve the 
retrieval process. And indeed, it has been seen that interesting retrieval behaviour 
emerges, especially for hnked patterns, e.g., different types of Mattis states occur, a 
higher retrieval transition temperature is possible, the quality of retrieval is improved 
through a bigger overlap. Therefore, it is interesting to investigate the properties of this 
model for infinite loading of patterns, i.e., for a ^ 0. This is the purpose of the present 
paper. 

After describing the ATNN-model in Section 2 we use the standard replica mean-field 
theory approach in order to write down its free energy in Section 3. Assuming replica 
symmetry we then obtain the fixed-point equations for the relevant order parameters. 
Thermodynamic and retrieval properties are discussed in Section 4. In particular. 
Section 4.1 investigates the behaviour of the model at zero temperature for equal two- 
and four-neuron coupling strengths in the Hamiltonian . In this case also the entropy 
is calculated and found to be negative. This leads to a discussion about the size of 
the replica-symmetry breaking and re-entrance behaviour. Section 4.2 presents phase 
diagrams in the temperature-capacity plane for different values of the two- and four- 



neuron coupling strengths. A number of new features in these diagrams show up. Also 
the maximal information content of the network is obtained as a function of these 
couplings. Where relevant, the results are compared with the Hopfield model [Q, ^ and 
with the four-state Potts model 0, |T0|, |1T|. Finally, Section 5 presents some concluding 
remarks. 



2. The model 

We consider a network of A^ sites each with two different types of binary neurons Si 
and ai,i = 1,...,N. The interactions between the neurons are determined by the 
infinite-range Hamiltonian 

H = - 
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In this network storage and retrieval of two different types of patterns, ^,j = {^f } 
and r7j = {r^f }, fi = 1, . . . ,p are enabled by a Hebb-like learning rule for the couplings 
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The patterns ^^ and 77^ are supposed to be independent identically distributed 
random variables (i.i.d.r.v.) taking the values +1 or —1 with equal probability. All 
Ju,^ = 1,2,3 are non-negative. 

First, we remark that the above definition of Jj- is equivalent to the case of linked 
patterns considered in [|l|. Second, we repeat that this model can be considered as 
an assembly of two Hopfield models (when J3 = 0) interconnected via a four-neuron 
interaction (when J3 7^ 0). 



3. Replica-symmetric mean-field theory 

In order to determine the thermodynamic and retrieval properties of the model we 
calculate the quenched free energy per site within the replica-symmetric mean-field 
theory approach. Using standard techniques we arrive at 
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with 13 = 1/T the inverse temperature measuring the noise level in the system, a 
the capacity defined as the number of patterns per number of couplings per spin, i.e.. 



a = 2p/3N and where 



a^ =irJ^m(;^^ + 2;^W-arJ , (4) 

K, = (3J,, < = e, ^^ = V', ^3^ = eV- (5) 

Here the index // = 1, ..., c, c finite, labels the condensed patterns, the index i/ = 1, 2, 3, 
((■ ■ ■)) indicates the average over the embedded patterns and Dz^ denotes the Gaussian 
measure Dz^ = dzj{2iT)~ ' exp{—z'^/2). In (3)-(4) the ni„ = {m(^} are the overlap 
order parameters between the pattern il)^, and the network state {5'^'^}, the g^ represent 
the Edwards-Anderson (EA) order parameters with their conjugate variables r^ (the 
mean-square random overlap with the non-condensed patterns), viz. 

^^ = ((^ E isn V',)) , ^^ = ((^ E iisnf)) > -^ = I t^ ((«)'» (6) 

with S} = Si, Sf = ai, Sf = Siai and where (■ ■ ■) denotes the thermal average. 

The phase structure of the network is determined by that solution of the set of 
fixed-point equations 
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which maximizes —(3f{(3). Here i/, 5, p = 1, 2, 3 and our convention is that they have to 
be taken different in the equations above. 

Solving this set of equations is a tedious task. In the following sections we discuss 
these solutions that are important for the thermodynamic and retrieval properties of 
the model, i.e., Mattis retrieval states and spin-glass states. At this point we remark 
that in the a = limit the replica-symmetric solution of the ATNN model presented 
above is completely consistent with the corresponding result obtained in m. 

4. Results 

The solution of the fixed-point equations for the ATNN neural network model presented 
in the previous section depends, of course, on the values of the coupling strengths K^. 
Without loss of generality we put Ki = K2 in the sequel. Such a choice is very often 
used in the literature concerning classical (|]12| and references therein) and spin-glass 
T3| Ashkin- Teller models. This model contains as particular limits the Hopfield network 



for any two K^, equal to zero, the four-state clock network |]T^ (or, equivalently, two 



independent Hopfield networks) for any K,^ = and a model strongly resembling the 
four-state Potts network |^, |10[ for Ki = K2 = K^. Remark, however, that only the 
strength of the couplings is the same but, in general, J^j 7^ J^j 7^ J^j (compare |p!3| ). 
Therefore, some details of the phase diagram compared with those of the Potts network 
may be different. 

The type of solutions we are interested in, at first instance, are Mattis solutions, for 
which not more than one component of each order parameter m^ is non-zero. As in [^ 
we thereby distinguish two types of Mattis states. Simple Mattis states where only the 
same components of the m,y are non-zero, e.g., mi = (mi, 0, ..., 0), m2 = (m2, 0, ..., 0), 
ma = (ma, 0, ..., 0), and crossed Mattis states where the same components of the m^, are 
never non-zero, e.g., mi = (mi, 0, ..., 0), 1112 = (0, m2, 0, ..., 0), m^ = (0, 0,m3, 0, ..., 0). 
These states are denoted by mim2m^ in the sequel. All these Mattis retrieval solutions 
also have non-zero values of the EA order parameters q^ . We remark that we have also 
found Mattis states where only one m^ and the corresponding q^ are non-zero. These 
are, in fact, Hopfield-like solutions. Furthermore, we are interested in spin-glass (SG) 
states characterised by ray = 0, g^ 7^ (here also Hopfield-like SG states are possible) 
and a paramagnetic state with all order parameters ra^ = 0, q^ = 0. 

First it is worthwhile to discuss the most simple case, i.e., zero temperature with 
equal coupling strengths. Second we present results for non-zero temperatures and 
arbitrary couplings. 

4.I. Retrieval properties at zero temperature 

Restricting ourselves first to Ki = K2 = K^ we find that only solutions with equal order 
parameters, mi = 1112 = ma and qi = q2 = qs exist. Further, considering simple Mattis 
states at zero temperature the set of fixed-point equations (7)- (9) can be reduced to a 
single equation, as in 





erf^ (^ + 2x) +erf' ( -5=) \l - zxV2\ - x\l ^ [l + e''"''^) (10) 



with the new variable x = m/v3ar being different from zero only when m 7^ 0. Thus 
the range in x allowing the existence of non-zero solutions of eq. (10) determines the 
critical capacity, ac(0), of the ATNN model with equal coupling strenghts at T = 0. 
Numerically solving this equation for different values of a we find adO) = 0.1839205. 
This is higher than the critical capacity of the Hopfield model (ac(0) = 0.137905566) 



T5| , p!6| and agrees (up to numerical precision) with the result obtained in ref. 0] for 
the four-state Potts neural network after an appropriate rescaling with the number of 
couplings as discussed in [IT^. (The relation between the Hopfield and Potts critical 
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capacities is given in eq. (19) of [|].) We remark that taking the couphng strengths K^, 
unequal always leads to a smaller critical capacity, as we will see in the next subsection. 



In analogy with the Ashkin- Teller spin-glass |T3[ and with the Hopfield network [^ 
we expect replica symmetry to be broken for low temperatures. Consequently, we expect 
some re-entrant SG-behaviour (see, e.g., [|TH]) indicating that the true critical capacity 
is greater than its replica symmetric value at T = 0. In order to get an idea about this 
breaking we have calculated the entropy of the replica-symmetric solution at T = 0, 
which for simple Mattis states reads 
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As expected we find that the value of this entropy is negative. In particular, S{ac{0)) = 
-0.007228 versus -0.001445 for the Hopfield model 0, -0.003995 for the three-state 
Potts model [^ and —0.007212 for the four-state Potts model |[ll[]. Furthermore, this 



entropy vanishes exponentialy with decreasing a and, in fact, for small a we have that 
S ~ — exp(— 1/«). In the SG-phase S = —0.91 at ac(0) versus —0.07 for the Hopfield 



model and —0.13 |T0[ for the three-state Potts model. All this suggests that replica- 
symmetry breaking of the retrieval states of the ATNN model is somewhat stronger 
than in the Hopfield model but still weak compared with the SG state of the ATNN. 
Further details on this matter are beyond the scope of the present work. 

4-2. The phase diagram for non-zero temperatures 

The phase diagram of the ATNN model is obtained by numerically solving the set of 
fixed-point equations (7)-(9). It is a complicated function of Ki = K2, K^, and a, the 
behaviour of which we analyse by looking specifically at T— a sections for different values 
of the ratio w = K1/K3. Furthermore, we discuss the maximal information content of 
the network in the full space of couplings. 

An extensive numerical analysis allows us to distinguish essentially two qualitatively 
different T — a sections of the full phase diagram. They are shown in figs. |I| and |[ 

Several transition lines bordering different phases show up. The lines T^ and Tj 
are related to simple Mattis solutions, the line Tg separates the SG and paramagnetic 
solutions. The lines T^o and T^m are connected with crossed states. Finally, the lines 
Tmi, Tgi and Tn have to do with the Hopfield-like retrieval and SG solutions introduced 
before. We remark that from now on writing that a solution exists also implies that it 
is stable within the replica-symmetric approximation, unless stated otherwise. 

Let us first discuss fig. |I| where the coupling strenghts are all equal, and hence 
w = 1, in more detail. We start looking at high T expressed in units of i^f ^ = T/Ji. 
The transition from the disordered paramagnetic phase to the SG-phase {m.i, = 0, 
gj, 7^ 0) is continuous and denoted by Tg. When crossing the line T^ simple Mattis 



retrieval states show up as local minima of the free energy. At these points the overlap 
with the embedded patterns jumps from zero to a finite macroscopic value. So the 
system functions as an associative memory and the critical storage capacity for a given 
temperature can be read off through that line. 

When further lowering T the simple Mattis states become global minima of the 
free energy. This happens along the line Tt and this thermodynamic transition is first 
order. Here we note that the critical lines T^ and Tg end in different temperature 
points at a = giving rise to a "crossover" region for small a as it occurs in the Potts 
model dTOf. This is related to the fact that for a = this ATNN model with w = 1 



has a discontinuous transition at Tf as shown in [||] (see in particular fig. 5). In this 
crossover region the simple Mattis states (global minima) and the paramagnetic state 
(local minimum) coexist. 

Finally, at still lower T (and appropriate values of a) crossed states of the type 
mmm and mmQ exist left from the line Tcm and T^o respectively. Their free energy is 
bigger than that of the simple Mattis states. 

We end the discussion of this phase diagram by remarking that we find weak re- 
entrant behaviour both at the lines Tt and T^. The re-entrance at the line Tt is somewhat 



stronger and may suggest [^ a greater replica-symmetry breaking of the SG-solution, 
in agreement with the values of the zero-temperature entropy calculated in section 4.1. 
Furthermore the greatest capacity of the ATNN model with equal coupling strengths is 
obtained for temperature T = 0.09 and reads adT = 0.09) = 0.1851. 

In figs. 0and|^ we present some typical overlap profiles for simple and crossed Mattis 
states in this model indicating the stability of the solutions within the replica- symmetric 
approximation (i.e., indicating where the solution is a minimum of the replica-symmetric 
free energy). We see that, for the same values of T and a the simple states always have 
the largest overlap. 

Next, we turn to a discussion of the phase diagram for the ATNN model with 
unequal coupling strengths. We explicitly consider the situation of fig. ^ where w = 1/4 
(meaning that the four-neuron term has four times the strength of the two-neuron one). 
Starting from high T we first meet the (new) line Tgi marking the continuous transition 
from the disordered paramagnetic phase to a SGl-phase where only one of the Qi, is 
non-zero, i.e. the Hopfield-like SG-state, m,y = 0, gi = g2 = 0, gs ^ 0. When lowering T 
and taking a sufficiently small the (new) line T^i appears below which solutions with 
only one m,^ and corresponding q^, different from zero, i.e., Hopfield-like Mattis states 
exist as local minima of the free energy. Below the thermodynamic first order transition 
line Tti they become global minima of the free energy. These Hopfield phases exist in 
the interval w = [0,0.69]. The rest of the lines which are present are similar to the 
ones of fig. [^ whereby we remark that the simple Mattis states are now of the form 
mml. However, crossing the line T^ from within the Hopfield-like retrieval phase we 



observe that the overlap mi = 1712 continuously increases from zero. Outside this region 
mi = 1712 immediately jumps from zero to a finite macroscopic value as is the case in 
diagram ^ Furthermore, crossed Mattis states of the type mml have become unstable 
for this value of w whereas the crossed states of type mmO exist for all w > 0. 

Sections of the full phase diagram for different values of the ratio w are qualitatively 
not much different and can be explained by taking the diagram fig. ^ with w = 1 as a 
reference. 

When w is increasing (starting at the value 1), meaning that the coupling strength 
of the four-neuron interaction becomes smaller, the crossover region shrinks since the 
endpoints of the lines Tg and T^ come closer. It finally shrinks to zero ioi w = 00 
meaning that one is left with two independent Hopfield models. The region where 
the crossed solutions mmO are found gets larger such that the line T^o moves to the 
right until it finally coincides with the line T^. The shapes of the transition lines are 
deforming continuously in such a way that they resemble more and more the Hopfield- 
model transition lines and finally they become identical with the latter. 

When decreasing w from the value 1 the phase diagram shown in fig. |l] evolves to 
the one presented in fig. |. The gap between the lines T^ and Tg for a = is increasing 
and Hopfield-like phases appear. When further decreasing w the gap still grows until it 
reaches the maximal value 1 in units of Ki^ and the Hopfield phase occupies a larger 
part of the phase diagram to become completely dominating in the limit w = where 
the pure Hopfield model is found. This means that the lines Tg, Tm and Tj shrink 
and disappear as w — *> and as a result the lines Tmi and Tn become equal to the 
corresponding lines of the pure Hopfield model. 

Finally, in order to have an idea about the amount of information stored in the 
network we have to take into account both the value of the capacity and the retrieval 
overlap. We follow the approach of |2^ based upon the Shannon entropy [^. This 
leads straightforwardly to the following information content per number of couplings 
per spin 



J(a,mi) = a 



-(1 + nil) ln(l + mi) + -(1 - mi) ln(l - mi) 



(12) 



We remark that only mi appears in this formula since for Ki = K2 we recall that 
mi = m,2. We have calculated /(a, mi) for all a < Uc and fig. 5 shows its maximal 
value, J, as a function of Ki^ and K^^ ■ A few remarks are in order. First, we find 
that the maximal information content is reached for values of a slightly smaller than 
Qc- Second, the greatest information, i.e. I = 0.1576, is obtained for equal coupling 
strenghts. Third, for w = 00 we find a constant information content independent of 
J3 which is equal to the one in the Hopfield model at zero temperature. This Hopfield 
value is 0.1213 (see |^), i.e., 0.0809 in fig. 5 in view of the rescaling with 2/3 due to 
the different number of couplings per spin. Finally, ioi w = and < Kf ^ < 2 the 



critical capacity is equal to the Hopfield value and zero for K^^ > 2 but the information 
content is continuously going to zero as K^^ tends to 2. This is due to the fact that the 
overlap mi = 7712 corresponding to maximal information goes continuously to zero. 

5. Concluding Remarks 

In this paper we have studied the thermodynamic and retrieval properties of the ATNN 
neural network model storing an infinite number of patterns using replica-symmetric 
mean-field theory. This analysis extends our results for this model in the case of low 
loading |jl|. 

We have derived the free energy and fixed-point equations for the relevant order 
parameters and have obtained the phase diagram for arbitrary values of the coupling 
strength of the two- and four-neuron interactions. This model can be considered as 
the sum of two interacting binary networks such that it contains the Hopfield model 
when some of the couplings go to zero. For equal coupling strengths this model strongly 
resembles the four-state Potts neural network. 

The storage capacity defined as the number of patterns per number of couplings 
per spin is maximal for equal coupling strenghts and equal to 0.1851. In that case 
the maximal information content per coupling is 0.1576 versus 0.1213 for the Hopfield 
model. 

Different retrieval characteristics can be distinguished in the phase diagram 
depending on the parameters of the network. First, the two types of embedded patterns 
are retrieved and the retrieval quality is enhanced by the interaction between the 
networks. These retrieval states are the simple Mattis states. Second, the two types of 
embedded patterns are retrieved but with lower precision because of the presence of this 
interaction. These retrieval states are the crossed states of the type mmO and mmm. 
They do not play an important role in the thermodynamics of the model. Third, a pure 
Hopfield-like phase is present indicating the retrieval of only one kind of pattern. 

These results allow us to mention some possible biological applications of this model. 
Patterns related with two different order parameters may be seen as a representation for 
two different stimuli, e.g., two visual ones as shape and colour, or two different ones as 
colour and flavour. When these two stimuli are somehow linked, e.g., through the fact 
that they were experienced at the same time, it is very likely that they will be recalled 
together, what corresponds to a suggestion in ref. |^ that particular combinations of 
stimuli are favoured. 

Finally, we remark that there is some re-entrant behaviour in the phase diagram at 
low temperatures which points at replica-symmetry breaking effects. Calculation of the 
zero-temperature entropy supports the expectation that this breaking is weak. 

In brief, the presence of a four-neuron coupling term in the ATNN model not 
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only enhances the quahty of pattern retrieval (by giving a larger overlap at higher 
temperatures), as present already in the case of low loading [Q, but it also increases the 
critical capacity and the maximal information content in comparision with the Hopfield 
model. 
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Figure 1. The T — a section of tiie piiase diagram for the ATNN model with equal 
coupling strengths (w — 1). The solid lines indicate the thermodynamic transitions. 
The precise meaning of the lines is described in the text. 
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Figure 2. Typical overlap profiles as a function of the temperature for simple 
Mattis states of the model with w = 1 for different capacities. From bottom to top 
a = 0, 0.09, 0.18. The dotted parts of the curves are unstable within replica symmetry. 



12 



K 



-1 



m 



0.93 z 



0.73 



0.53 



0.33 
0.000 




0.005 



0.010 



a 



Figure 3. Typical overlap profiles as a function of the capacity for crossed Mattis 
states mmO of the model with w = I for different temperatures. From top to bottom: 
T = 0.5, 0.55, 0.58, 0.6, 0.62, 0.65. The dotted parts of the curves are unstable within 
replica symmetry. 
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Figure 4. The T — a section of the phase diagram for the ATNN model with unequal 
coupling strengths (w = 1/4). The solid lines indicate the thermodynamic transitions. 
The precise meaning of the lines is described in the text 
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Figure 5. The maximal information content of the ATNN model as a function of K^ 



and K^ ^. 



